Skip to content

Portable DynamoDB and celld record providers on Fastly - #522

Draft
ChristianPavilonis wants to merge 12 commits into
feat/portable-client-infofrom
feat/474-portable-record-providers
Draft

ChristianPavilonis wants to merge 12 commits into
feat/portable-client-infofrom
feat/474-portable-record-providers

Conversation

@ChristianPavilonis

Copy link
Copy Markdown
Collaborator

Summary

  • Run the DynamoDB and celld record providers from Add portable conditional KV records across providers #510 on WASM targets over EdgeZero outbound HTTP, and prove them on Fastly. Native Axum transports keep their behavior; Fastly KV, Cloudflare KV and Spin KV stay native.
  • Fastly selects a provider per KV ID from a runtime bindings document with pinned identities written by an offline edgezero config bindings pin. Credentials come from a logical Secret Store with derived key names, read once per request.
  • Stacked on Add portable ClientInfo for Fastly, Axum, Cloudflare and Spin #517 (feat/portable-client-info, 10ed3803). Design: docs/superpowers/specs/2026-11-19-portable-record-providers-design.md; plan and evidence: docs/superpowers/plans/2026-11-19-portable-record-providers.md.

Changes

Crate / File Change
edgezero-store-aws Transport-independent record core (rows, cursors, validation, classifier). Native SDK transport unchanged in behavior. New dynamodb-records-portable: JSON 1.0 over HttpClient, signed with aws-sigv4, credentials from KV__<ID>__AWS_*. aws-dynamodb-records settings gain optional endpoint, pinned and credentials; native pinned preparation and preflight. Settings clocks use web_time.
edgezero-store-celld Request/response policy moved to a shared wire module used by the native reqwest client and a portable HttpClient exchanger. New portable-records: authenticated HTTPS only, pinned instance, token from KV__<ID>__CELLD_TOKEN. getrandom/wasm_js only on wasm32-unknown-unknown.
edgezero-adapter New store-bindings feature: the shared strict version = 1 document and an offline portable validator.
edgezero-adapter-axum Uses the shared document. Rejects credentials, token_secret and the DynamoDB endpoint; verifies pinned at startup. Real-service interop tests (tests/portable_interop.rs, ignored by default).
edgezero-adapter-fastly Features aws-dynamodb-records, celld-records. Reads edgezero_store_bindings / EDGEZERO__STORE_BINDINGS when the app is built, keeps prepared providers on the App, binds them per request over the effective HttpClient. Unbound IDs stay on Fastly KV; bad documents or uncompiled providers fail assembly, no fallback. Generated crate and fastly.toml templates forward build features.
edgezero-cli config bindings pin --adapter fastly --store-bindings <file> (feature bindings-pin, default on; writes only pinned, atomically). config validate --adapter fastly --store-bindings offline checks. Manifest build features reach provider-CLI builds via EDGEZERO_BUILD_FEATURES. Generated CLI template mirrors the command.
edgezero-test-harness portable-record-contract suite with strict per-provider receipts; portable fixture app on the canonical Fastly path; record cleanup drains every page; reviewed ownership fingerprints updated for the new selector and dispatch.
scripts/ run_portable_record_contract.mjs orchestrator (DynamoDB Local, official celld behind TLS, pin, native seed, Viceroy, interop), fetch_official_celld.mjs, dependency contract extended to portable and Fastly graphs.
.github/workflows/test.yml Portable WASM compile checks and a new job, "Portable record contracts (Fastly)".
docs/guide/* Fastly bindings, credentials, pin/validate/upload, IAM policy, budgets and error meanings.
Lockfiles Root, app-demo, geo-kv-prototype, harness fixtures.

Closes

Closes #474

Unblocks #479 and #484, whose runtime wiring is out of scope here.

Test plan

  • cargo test --workspace --all-targets (RUST_TEST_THREADS=1)
  • cargo clippy --workspace --all-targets --all-features -- -D warnings, plus Fastly cli/no-default, harness and fixture Clippy
  • cargo fmt --all -- --check (root, harness, fixtures)
  • cargo check --workspace --all-targets --features "fastly cloudflare spin"
  • WASM builds: portable providers on wasm32-wasip1, wasm32-unknown-unknown, wasm32-wasip2; Fastly adapter with both provider features on wasm32-wasip1; Spin check
  • examples/app-demo workspace: cargo test --workspace --all-targets
  • Docs build: npm run lint && npm run format && npm run build
  • Manual testing via edgezero serve --adapter axum
  • Other:
    • Harness tests 682; store tests 113 (AWS) and 47 (celld), including portable wire, record contract, native/portable row and cursor interop, lazy credentials, no-retry classification and the shared two-call deadline.
    • Full orchestrator in pinned Viceroy 0.21.0 against DynamoDB Local 3.3.1 and official celld 0.6.3 behind TLS: for each provider the shared contract once, independent Add and CAS each with one winner and stored-winner readback, 1001 descendants over several pages, native-row read/CAS/delete, cleanup to zero keys. Host interop 2/2, wrong-CA rejection 1/1, CLI preflight pins both providers.
    • Native runners on the final state: DynamoDB Local and official celld generated consumers and record checks pass.
    • scripts/check_native_store_dependency_contract.sh and Node tests pass.

Known, not caused here:

  • The existing native Fastly record-contract suite still fails in Viceroy 0.21.0 (both racing writers succeed), as recorded in the earlier record plan. The portable suite is separate so it neither fails nor hides that result.
  • Running the whole CLI library suite with --no-default-features --features cli,edgezero-adapter-axum shows 73 failures, identical on the base commit. CI runs only the bindings subset there (14 pass).

Open before merge:

  • Real Fastly p50/p95 for a conditional write against us-east-1 (P11). Needs a scratch Fastly service; docs mark it pending, no numbers recorded.
  • Mark "Portable record contracts (Fastly)" as a required check in branch protection.
  • Not executed at the Fastly level: credential rotation across requests on a retained App and a hook-supplied HttpClient (Viceroy cannot change secrets mid-run). Store-level tests cover the same logic.

Notes for review:

  • The portable runtime addresses DynamoDB tables by name and does no identity discovery. Recreating a bound table in place discards allocator rows; the docs say not to, and to re-pin if unavoidable.
  • The default CLI now includes the AWS SDK for the pin command. The SDK-free minimal build is unchanged and checked by the dependency contract.
  • Build-feature forwarding was checked with Fastly CLI 15.1.0 (sh -c expansion); the repo pins 16.0.0.

Checklist

  • Changes follow CLAUDE.md conventions
  • No Tokio deps added to core or adapter crates (portable graphs carry only Tokio sync via aws-smithy, allowed by the existing feature-aware check)
  • Route params use {id} syntax (not :id)
  • Types imported from edgezero_core (not http crate)
  • Store wiring goes through KvRegistry / ConfigRegistry / SecretRegistry (not the legacy single-handle setters) — see spec §6.6
  • New code has tests
  • No secrets or credentials committed (fixtures use fake keys; generated test tokens are per run and not kept as evidence)

Move row encoding, cursors, validation and the mutation classifier into a
transport-independent record core. The native SDK transport keeps the same
requests and deadlines. The portable transport sends DynamoDB JSON 1.0 over
EdgeZero outbound HTTP, signed with aws-sigv4 from per-request credentials.

Add the aws-dynamodb-records binding settings with an optional endpoint,
pinned table identity and logical credentials reference, plus native
preflight and pinned preparation for the offline pin command.
Move request headers and response validation into a transport-neutral wire
module used by the native reqwest client and a portable HttpClient
exchanger. The record store is generic over the exchanger. Portable bindings
require authenticated HTTPS and a pinned store instance, and read their
bearer token from a logical secrets store once per request.

Add celld-records binding settings, native preflight and pinned preparation.
Move the strict document and provider dispatch into a settings-only
edgezero-adapter store_bindings module. Axum keeps its file input and native
source selection, rejects portable credential references and the DynamoDB
endpoint, and verifies pinned identities during native preparation.

Add an offline portable validator for targets such as Fastly, plus native
pinned-preparation, preflight and portable celld exchanger tests.
Extend the native store dependency contract to the portable provider features
on all three WASM targets, including where aws-sigv4 may appear. Add
compile-only CI checks for both portable features and a verified download of
the pinned official celld 0.6.3 release for CI.
The offline pin command and the runtime providers share one derivation.
Fastly reads the optional edgezero_store_bindings Config Store document once
when the app is built, validates it offline and keeps prepared providers on
the retained App. Each bound KV ID is served by its portable provider over the
request's effective HttpClient; unbound IDs stay on Fastly KV. A bad document
or a provider missing from the binary fails assembly without fallback.

Add the aws-dynamodb-records and celld-records adapter features, forward
[adapters.fastly.build] features to generated Fastly builds, and add
'edgezero config bindings pin' plus offline Fastly bindings validation.
…ceroy

Add a portable fixture app that serves both KV IDs through the adapter's
registry, and a portable-record-contract harness suite with strict per-provider
receipts for the shared contract, independent Add/CAS races, 1001-key pages,
native-to-portable reads and cleanup. Cleanup now drains every global page.
Add real-service native and portable interop tests and an orchestrator that
provisions DynamoDB Local and official celld behind TLS, pins the bindings,
seeds native rows and runs the Viceroy suite. Check the Fastly provider
dependency graphs and add the required portable record contracts job.
The shared record contract asserts the exact global listing, so the natively
written interop row must be consumed first.
@ChristianPavilonis ChristianPavilonis self-assigned this Nov 20, 2026
] {
fs::write(&bindings, input).unwrap();
let error = run_config_validate(&args).unwrap_err();
assert!(error.contains(expected), "{error}");
@aram356
aram356 marked this pull request as draft November 20, 2026 17:12

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Portable DynamoDB and celld providers over EdgeZero's outbound HTTP, verified on Fastly

2 participants